System and method for controlling luminance during video production and broadcast

ABSTRACT

Disclosed herein are systems and methods for controlling luminance during video production and broadcast. An exemplary system includes a camera to capture video content in a first imaging range, a histogram calculator to evaluate luminance in each pixel in the captured video content, and to generate a luminance histogram for the captured video content, a user interface generator that generates a user interface displaying the video content overlaid with the luminance histogram and generates a user interface displaying a light intensity curve and adjustable parameters for converting the first range into a second range, a luminance controller to convert luminance of the video content into the second imaging range based on the light intensity curve, and a broadcast controller to encode the video content with the second imaging range into broadcast range for transmitting to one or more endpoint devices.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/631,397, filed on Feb. 15, 2018, the content of which is hereinincorporated by reference in its entirety.

FIELD

The present disclosure generally relates to an apparatus that assists incontent conversion in video production, and particularly to a system andmethod for controlling luminance during video production and broadcast.

BACKGROUND

High Dynamic Range (“HDR”) is a relatively new technology beingintroduced by a number of standard organizations, such as Blu-ray DiscAssociation, ISO/IEC HEVC, ITU-R, SMPTE, CEA, and HDMI, as well asprivate companies, such Dolby and Philips. HDR displays provide asignificant improvement over current display technology, such StandardDynamic Range (“SDR”) displays and monitors. In particular, HDR devicesproduce a peak brightness that is an order of magnitude larger than SDRtechnology and a dynamic range that approaches the capabilities of ahuman observer. Additionally, these devices provide an enlarged colorgamut and can reduce the motion blur in LCD systems, for example.

Current SDR devices do not support HDR content as such SDR devices aredesigned for current display technology and built on the assumption thatHDR content cannot be reproduced. Accordingly, in the current videoconsumption environment, there is a need to convert content from HDR toSDR in order to enable content playback on such SDR devices. Thisconversion from HDR to SDR will be a key technology while the newHDR-capable display monitors and televisions begin to penetrate themainstream consumer market in the United States and the rest of theworld.

HDR to SDR conversion will be a particularly critical technology in thevideo production and broadcast environment, and, specifically, for livevideo production and distribution. Currently, video production camerasused to capture live content are typically HDR enabled cameras.Moreover, camera shading operators located at the live event canmanually adjust the camera settings during content capture to accountfor brightness levels in view of the downstream HDR to SDR conversionthat will be performed to enable quality playout of the captured contenton the video consuming end-devices, such as SDR-capable display monitorsand televisions. However, despite camera shading operators' ability toaccount for brightness levels for video content capture, a system andmethod is needed to facilitate and enhance the HDR to SDR conversion forlive video production and video content broadcast.

SUMMARY

Thus, according to an exemplary aspect, a video production system isdisclosed that includes one or more user interfaces for displayingcontent and user controls for managing the HDR to SDR conversion processfor video distribution and broadcast.

In general, the system includes at least one, and more likely aplurality of HDR enabled cameras, located at a live event and providedfor content capture. The HDR content will then be encoded and streamedto a production environment, such as a production truck or videoproduction center, where the HDR content will be processed before videodistribution and broadcast. In one aspect, the video contentdistribution signal will comprise a simulcast signal in which onechannel distributes/broadcasts the content in HDR while a second channeldistributes/broadcasts the content in SDR. In another aspect, thebroadcasted signal will be a unicast signal in SDR (converted from thecaptured HDR content). The mechanisms for content distribution are notcritical to the system and method disclosed herein and will not bedescribed in detail.

In an exemplary aspect, the video production system provides two userinterfaces that enable the camera shading operator (or similar videoproduction user/operator) to manage and control the intensity of thecaptured content for the HDR to SDR conversion. More particularly, afirst user interface (e.g., a content zone selection and histogramgeneration interface) enables the user to select a zone or region (e.g.,a defined rectangular zone and referred to as a “snapshot”) of thecaptured content to generate a histogram (i.e., a “nits histogram”) thatmeasures the nits levels of the selected region of the captured content.It is noted that a “nit” is a unit of luminance equivalent to onecandela per square meter. Thus, in this aspect, this first interfaceenables the user to select and modify the zone/region of the capturedcontent that is received to calculate the nits histogram (e.g., ameasured luminance level histogram) of the captured HDR content.

The second interface (e.g., a light intensity curve adjustmentinterface) enables the operator to visualize the algorithm's parameter'seffect on the tone mapping function and the snapshot area. The secondinterface is configured such that the operator (e.g., the camera shadingoperator) can manually adjust the intensity curve (e.g., a Bézier curve)based on a plurality of displayed parameters to control the HDR to SDRconversion. It is noted that the term “tone mapping” refers to thetechnique for the video processing described herein for mapping one setof colors (i.e., the HDR content) to another set of colors (i.e., theSDR content) to approximate the appearance of the HDR images in thevideo consumption devices, i.e., the SDR-capable display monitors andtelevisions that have a more limited dynamic range as described above.

In accordance with an exemplary embodiment, a system is provided forcontrolling luminance of video content during video production. In thisembodiment, the system includes at least one camera configured tocaptured video content in a first imaging range; an analysis regionselector configured to select at least one region in at least one frameof the captured video content, the selected at least one regioncomprising a plurality of pixels; a pixel luminance evaluator configuredto measure a respective luminance level for each of the plurality ofpixels in the selected at least one region in the at least one frame ofthe captured video content; a luminance histogram generator configuredto generate luminance histogram based on the measured luminance levelsof the plurality of pixels in the selected at least one region; and afirst user interface configured to display at least one frame of thecaptured video content and the generated luminance histogram as anoverlay on the displayed at least one frame. In addition, the exemplarysystem further includes a second user interface configured to display alight intensity curve relative to a plurality of adaptive parametersdefined based on the luminance values of the generated luminancehistogram; a luminance controller configured to convert the capturedvideo content in the first imaging range to broadcast video content in asecond imaging range, different than the first image range and having aluminance based on output luminance values of the light intensity curve;and a video content distribution controller configured to encode thebroadcast video content in the second imaging range for distribution toat least one end consuming device for display thereon.

Moreover, in a refinement of the exemplary embodiment, the first imagingrange is a high dynamic range and the second imaging range is a standarddynamic range and the captured video content is live video content.Furthermore, in an exemplary aspect, the adaptive parameters displayedin the second user interface include an SDR reference range, an SDR peakwhite, a tonemap Max-Destination, a tonemap HDR range, and an HLGmonitor. Yet further, the analysis region selector can include at leastone first control input configured to receive an operator input toselect a size and a position of the at least one region in the at leastone frame of the captured video content. Moreover, the second userinterface can include at least one second control input configuredadjust a position of the light intensity curve relative to the pluralityof adaptive parameters to set the output luminance values for theluminance controller to convert the captured video content in the firstimaging range to broadcast video content in the second imaging range.

In accordance with another exemplary embodiment, a system is providedfor controlling luminance of video content during video production, thesystem comprising, at least one camera configured to capture videocontent in a first imaging range, a histogram calculator configured toevaluate luminance in the captured video content, and generate aluminance histogram for the captured video content, a user interfacegenerator configured to: generate a first user interface displaying thecaptured video content overlaid with the generated luminance histogram,and generate a second user interface displaying a light intensity curveconfigured with parameters for converting the capture video content fromthe first imaging range into a second imaging range; and a luminancecontroller configured to convert the captured video content into thesecond imaging range based on the parameters of the light intensitycurve.

In accordance with yet another exemplary embodiment, a system isprovided for setting luminance of video content, the system comprising:a histogram generator configured to generate a luminance histogram basedon measured luminance values for at least one region in video content ina first imaging range, a user interface generator configured to generateat least one user interface configured to display the generatedluminance histogram a light intensity curve with parameters forconverting the video content from the first imaging range to a secondimaging range and a luminance controller configured to convert the videocontent to the second imaging range based on settings of the parametersof the light intensity curve.

The above simplified summary of example aspects serves to provide abasic understanding of the present disclosure. This summary is not anextensive overview of all contemplated aspects, and is intended toneither identify key or critical elements of all aspects nor delineatethe scope of any or all aspects of the present disclosure. Its solepurpose is to present one or more aspects in a simplified form as aprelude to the more detailed description of the disclosure that follows.To the accomplishment of the foregoing, the one or more aspects of thepresent disclosure include the features described and exemplary pointedout in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for generating SDR video contentfrom HDR video content in accordance with an exemplary embodiment.

FIG. 2 illustrates a block diagram of the histogram calculator forgenerating SDR video content from HDR video content in accordance withan exemplary embodiment.

FIG. 3 illustrates an exemplary user interface for generating SDR videocontent from HDR video content in accordance with an exemplaryembodiment.

FIG. 4 illustrates an exemplary histogram for generating SDR videocontent from HDR video content in accordance with an exemplaryembodiment.

FIG. 5 illustrates another exemplary user interface for generating SDRvideo content from HDR video content in accordance with an exemplaryembodiment.

FIG. 6 illustrates a flow diagram for a method for controlling luminanceduring video production and broadcast, in accordance with exemplaryaspects of the present disclosure.

FIG. 7 is a block diagram illustrating a computer system on whichaspects of systems and methods for controlling luminance during videoproduction and broadcast may be implemented in accordance with anexemplary aspect.

DETAILED DESCRIPTION

Various aspects of the disclosure are now described with reference tothe drawings, wherein like reference numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to promotea thorough understanding of one or more aspects of the disclosure. Itmay be evident in some or all instances, however, that any aspectsdescribed below can be practiced without adopting the specific designdetails described below. In other instances, well-known structures anddevices are shown in block diagram form in order to facilitatedescription of one or more aspects. The following presents a simplifiedsummary of one or more aspects of the disclosure in order to provide abasic understanding thereof.

FIG. 1 is a block diagram of a system 100 for generating SDR videocontent from HDR video content in accordance with exemplary embodimentsof the present disclosure. As shown, the system 100 comprises an imagecapture device 102 (e.g., a camera), an image processor 110, an operatordisplay 124 and one or more endpoint displays 150 (e.g., contentconsuming devices, such as a computer or television, for example).

The capture device 102 is configured to capture HDR video content 104and delivers the video content to the image processor 110. The HDRcontent 104 is processed by the image processor 110 that ultimatelygenerates SDR content 141 for output to the one or more endpoint devices(e.g., SDR displays) such as device 140. The image processor 110generates a user interface for an operator that allows the operator toadjust the conversion from HDR to SDR dynamically during live or videocontent broadcasts. In some aspects, the generated user interfaces arepartially based on endpoint device display information 129. In order toadequately generate the SDR content 141, the image processor receivesadjustments by the operator and adjusts the generation processaccordingly, specifically targeting a luminance curve of the generatedcontent. It should be appreciated that HDR to SDR conversion aredescribed as the exemplary imaging ranges, but it is contemplated thatthe disclosed invention can be used to convent image content betweendifferent types of content.

In exemplary aspects, the image processor 110 comprises a histogramgenerator 112, a parameter tracker 114, a luminance controller 116, auser interface (UI) generator 118, a broadcast controller 120 and aregion selector 122. The image processor 110 receives the HDR content104 as content is being broadcast live or at a later time.

Initially, the UI generator 118 is configured to generate a userinterface 130 shown in FIG. 3 to display on an operator display 124. Inthe UI 130, the operator is presented one or more frames of the HDRcontent 104. Using the UI 130, the operator can select one or moreregions of the frame, e.g., region 300, by using a mouse or touchscreenfor example, with the region 300 representing a variation in theluminance that the operator wishes to preserve in an SDR conversion ofthe HDR content 104. The region selector 122 is configured to capturethis region and transfer control to the histogram generator 112.

Once this region is selected, the histogram generator 112 is configuredto evaluate the light intensity of each pixel contained with theselected region 300, and generate a histogram of the light intensity ineach of the pixels. For example, FIG. 2 illustrates a block diagram ofthe histogram calculator that receives HDR content 104, and based on theluminance in each pixel, sorts the pixels into bins 1 to N. The bins 1-Nare used by the UI generator 118 in the generation of the userinterfaces 130-132. In an exemplary aspect, the calculated histogram400, as shown in FIG. 4, is overlaid on the content being currentlyprocessed by the image processor 110. It is noted that the content hasbeen omitted from FIG. 4 for the sake of clarity. The histogram 400shown in FIG. 4 is, for example, a histogram of a frame containingpixels that are mostly black or near black.

The UI generator 118 is also configured to generate a user interface132, shown in FIG. 5, for the conversion of the imaging content (e.g.,HDR content) from its first imaging range to a second imaging range(e.g., SDR). The parameter tracker 114 is configured to capture changesmade to various parameters adjustable by the operator in the userinterface 132. In some aspects, these parameters include “HDR MAX”, “SDRMAX”, “SDR TMmd”, “SDR reference range”, “SDR peak white”, “Max. Dest.”,“tonemap HDR range” and “HLG monitor”, though other parameters are alsocontemplated herein. These parameters will be discussed in furtherdetail below. In exemplary aspects, the operator may adjust theseparameters via controls on the display 124, touch screen, or any othertype of input device. As the operator adjusts the parameters, the lightintensity curve 500 is generated. The light intensity curve 500 dictatesthe conversion from the light intensity range of the HDR content 104 toa second light intensity range that will be used in the SDR content 141to maintain adequate light levels in endpoint devices.

Once the parameters are adjusted by the operator, the luminancecontroller 116 is configured to apply the changes to the HDR content 104to generate SDR content based on the light intensity curve 500. Finally,the broadcast controller 120 encodes the generated SDR content,partially according to the endpoint display information 129 andtransmits the SDR content 141 to endpoint devices such as display 150.

FIG. 4 illustrates the luminance histogram 400 according to an exemplaryembodiment.

It is noted that the exemplary luminance histogram shown in FIG. 4corresponds to an image with a black background, in order to adequatelydescribe the display of the histogram 400 for purposes of thisdisclosure. However, according to an exemplary aspect, the calculatedhistogram 400 and adaptive parameter user interface 132 may be presentedas an overlay on top of portions of the captured content 104. In otherwords, the system 100 can include a video monitor (in the productiontruck, for example display 124) that displays the captured HDR content104 and generates the luminance histogram 400 that is overlaid on thecaptured content to provide the user interface 132.

According to the exemplary aspect, the user interface 131 is configuredto generate and display the luminance histogram 400, which displays theproportion of pixels in each luminance range. As shown above, in theexemplary aspect, the luminance histogram 400 includes 16 separateluminance ranges with the first range beginning at less than 0.5 nits(i.e., “<0.5” nits) and the last range being for pixels in the8192-10000 nits range (i.e., “10000” nits). Thus, as shown, theluminance histogram 400 includes 16 separate “bins” for displaying theluminance of pixels in each range.

According to the exemplary aspect, the system 100 includes a luminancehistogram generator 112 (which can be a software module executed by acomputer processor) that is configured to calculate the nits histogramby precomputing a 1D LUT (i.e., a one dimensional lookup table) wherethe key is X bits, a sum of n, m and o, where n is msb (most significantbits) of Y′, m is msb (most significant bits) of Cb′, o is msb (mostsignificant bits) of Cr′. The LUT value is the NITs for that key X. Inthis aspect, n, m, and o are numeric values corresponding to the numberof bits of the pixel's Y′, Cb′ and Cr′ components (respectively) thatcontribute to the LUT key. The histogram generator 112 also generatesthe histogram for use in the user interface 131. For each analyzedpixel, the LUT directly generates a bin index (or luminance range), andthe histograms bin's counter is incremented accordingly.

In one exemplary aspect, a predefined limit can be selected to show thepercentage of pixels that fall into a particular luminance range. Forexample, if 40% (as an exemplary predetermined limit) of pixels fallinto a particular luminance range, that histogram bin will be shown tohave a full bar. For example, in the example of FIG. 4, since thehistogram 400 represents a black image, every pixel will have a nitsvalue of less than 0.5. Therefore, since the percentage of pixels isgreater than 40% (i.e., it is 100% in this example), the <0.5 nits binis shown to have a full bar. It should be appreciated that when anactual image of captured content 104 is analyzed to calculate thehistogram, each bin will have varying levels based on the actual pixelsthat fall within each luminance range.

As further shown, the user interface 131 is configured to derive anddisplay adaptive parameters that enable the operator to controlhighlights and midtones separately. Specifically, in the exemplaryaspect, these parameters includes “SDR Max” and “HDR Max”. The HDR Maxparameter is a parameter value that drives the conversion from SDR toHDR. The HDR Max parameter affects highlight compression, trading offluminance of the brightest pixel in an image for greater detail in thebrightest parts of the image. In adaptive mode, analysis of the capturedimage may drive changes to HDR Max. If this value is low then, then theproduction system will perform less highlight compression to preservehighlight granularity. Moreover, the SDR Max is the Max SDR value, whichindicates where to map the SDR diffuse white point. In an exemplaryaspect, proper control of this SDR Max parameter is a key factor incontrolling the image conversion, i.e., the system 100 ensures that mostof the pixels in the captured image are below the diffuse white point.In the exemplary aspect shown in FIG. 4, the SDR Max value is set to 203nits and is configured to be increased if the weighted average of thehistogram bins goes above this value. Furthermore, the parameter “SDRTMmd” indicates the SDR Tone Mapping max-destination. It should beappreciated that each of the parameter terms illustrate in FIG. 4 areselected for an operator's perspective to represent parameters of thesource image and capabilities of the target display.

Moreover, the user interface is configured to display a parametertracker, which in the exemplary aspect is provided as graphs 402 in thelower right of the interface 131. In the example of FIG. 4, the twolines on the lower right are trackers for the HDR Max and SDR Maxparameters and graph the recent values of these two parameters. In thisexample, the parameter trackers are shown as flat or horizontal lines,which indicates that the parameter have not recently been changed in thecaptured frames of the HDR content. In operation, the parameters willchange upon each scene change, frame change, etc.

As noted above, the first user interface 130 is configured to enable theoperator to select a snapshot (i.e., a region or zone) that is used forthe histogram calculation. In some exemplary aspects, the first userinterface 130 may be combined with the user interface 131, allowing theoperator to select the snapshot (the one or more regions to be analyzed)as the histogram is concurrently displayed. In this aspect, thehistogram will change as the operator's selection changes.

As shown in FIG. 3, the user interface 130 provides an image with aselected region 300 that can be used to calculate the luminancehistogram. Those of ordinary skill in the art will recognize that whilean exemplary aspect of the present disclosure region selection can beperformed on portions of an image. However, in another exemplary aspect,the histogram is calculated for the entire frame as opposed to aselected region. In one aspect, to select a region, the monitor caninclude one or a plurality of knobs that enable the operator to selectthe size and position of the snapshot. For example, a first knob enablesthe user to zoom in or out in a particular region thereby adjusting thesize of the selected region. A second knob enables the operator toadjust the position (e.g., moving a cross point according to the X and Yposition within the image). By doing so, the operator can select theparticular region of the image, and, more particularly, the pixelswithin the defined region, such that the luminance values for each pixelwithin the region are used to calculate the nits values for theluminance histogram, which is then presented as an overlay as describedabove with respect to FIG. 1. Advantageously, by enabling the operatorto define the particular region with the captured HDR content, theoperator can effectively select regions of the image that may mostsignificantly affect the luminance when the HDR content is converted toSDR format for content distribution as discussed above.

Moreover, it is noted that while the exemplary aspect describes the useroperating inputs as knobs configured to control the size and position ofthe luminance histogram region, the size and position can be adjustedaccording to other possible user inputs. For example, the user interface130 can be provided on a touchscreen with the user being able to defineand adjust the region or zone directly with touch inputs on the screenas would be appreciated by one skilled in the art. Moreover, in anexemplary embodiment, the default setting is that the histogram isgenerated for the entire frame and that no specific region is selected.

In either case and as further described above, a second interface 132(e.g., a light intensity curve adjustment interface) can also beprovided to an operator (e.g., the camera shading operator) in theproduction environment on display 124, which enables the operator toadjust the intensity curve 500 (also referred to as a tone mappingcurve) with relation to parameter values that are defined according tothe luminance histogram created for the captured HDR content, asdescribed above.

FIG. 5 illustrates a screenshot of the user interface 132 according toan exemplary aspect.

In general, the exemplary system and method described herein isconfigured to perform an HDR to SDR tone mapping algorithm for theconversion and video content production process. In general, the HDR toSDR tone mapping performs color conversion and HDR to SDR tone mapping.Moreover, this conversion must balance the preservation of artisticintent with the technical limitations of SDR and Rec-709 color space,for example. Advantageously, the tone mapping algorithm utilizesparameters that will automatically control the trade-off betweenpreserving detail in very bright areas of the HDR content and preservingan appropriate level for middle and dark tones to ensure that the SDRvideo production has acceptable levels of illumination and contrast froman end user perspective.

According to the exemplary aspect shown in FIG. 5, the tone mappingcurve 500 (e.g., a Bézier curve) is generated and displayed, enablingthe operator to visualize the algorithm's parameters' effect on the tonemapping function and the region 300 shown in FIG. 3 (i.e., the selectedluminance region of the HDR content as discussed above). In this aspect,the X axis is the input luminance (i.e., the luminance of the HDRcontent) and the Y axis is output luminance (i.e., the luminance of theSDR content).

Moreover, the exemplary light intensity curve adjustment interface(i.e., second user interface 132) generates a plurality of parametervalues that enable the operator to fit the tone mapping curve for theSDR conversion. As shown, the parameter values include: “SDR referencerange”, “SDR peak white”, “Max. dest.”; “Tonemap HDR Range”; and “HLGMonitor”. It is noted the each of the initial conditions for theparameter values are set by the capabilities of the target monitor asreceived in the endpoint display information 129, the luminance encodingor transfer characteristics of the source data, and the measured resultsof the frame, or video block, being analyzed for mapping.

According to the exemplary aspect, the parameters shown in this userinterface 132 are adjusted based on the luminance values of the selectedregion 300 of the HDR image, which are represented by the luminancehistogram 400 as described above. For exemplary purposes, fourmethodologies are provided, which include: (1) High Mid-tone DetailEnhancement; (2) Highlight Detection; (3) Strong Highlight Enhancement;and (4) Gamut excursion correction. In some aspects of the disclosure,the user interface 132 may include graphical elements for adjusting eachof the adjustable parameters, and for selecting one or more of thesemethodologies. The operator may select one or more of thesemethodologies in the user interface 132 using the graphical element, or,for example a physical knob, or the like, and the adjustable parametersmay automatically be adjusted to values that produce a light intensitycurve based on the selected or disabled methodologies.

In an exemplary aspect, High Mid-tone Detail Enhancement involvesraising the SDR peak white value shown in FIG. 5. For example, ifanalysis of the luminance histogram 400 indicates a large portion (e.g.,a percentage above a predetermined threshold) of midrange and lowhighlight pixels (e.g., between 4 and approximately 1000 nits) arehighlights (e.g., >˜250 nits), the system 100 is configured to raise theconversion algorithm's SDR peak white parameter value by a relativevalue in order to allocate enhanced resolution for these pixels.

In an exemplary aspect, Highlight Detection involves reducing theconversion algorithm's tonemap HDR Max parameter (i.e., the “Tonemap HDRrange” shown in FIG. 5 above). For example, if analysis of the luminancehistogram 400 indicates that very few pixels (e.g., a percentage below apredetermined threshold) are highlights (e.g., above approximately 500nits), the system 100 is configured to reduce the conversion algorithm'stonemap HDR Max parameter in order to enhance the luminance of pixelsthat ordinarily would be white or near-white on an SDR display, butwould otherwise be reduced in intensity to illustrate even brighterpixels in the SDR signal.

In an exemplary aspect, Strong Highlight Enhancement involves raisingthe conversion algorithm's tonemap HDR Max parameter (i.e., the “TonemapHDR range” shown in FIG. 3 above) and the tonemap Max-Destinationparameters (i.e., the “Max. dest.” shown in FIG. 5 above). For example,if analysis of the luminance histogram 400 indicates that a significantportion (e.g., a percentage above a predetermined threshold) of pixelsare very bright (e.g., luminance above 2048 nits), the system 100 isconfigured to raise the conversion's tonemap HDR Max parameter andTonemap Max-Destination parameters in order to linearize the highestportion of the Bézier down-mapping curve used in the algorithm (i.e.,the tone mapping curve shown in FIG. 3), which allows for more detail toemerge in the very strong highlights.

In an exemplary aspect, Gamut excursion correction involves raising theSDR Ref White parameters (i.e., the “SDR reference range” shown in FIG.5 above). For example, for certain HDR content, a significant portion ofHDR pixels (e.g., 2020 HDR pixels) are not properly expressible in theSDR 709 space, which would result in very “patchy” images with severeloss of detail. This technical effect is possible due to high luminancepixels being outside the 709 gamut. Therefore, when analysis of theluminance histogram 400 indicates that this condition existing for thecaptured HDR content, the system 100 is configured to lower the overallluminance by raising SDR Max, to effectively bring these out-of-gamutpixels into range and restore detail.

It is reiterated that each of these four methodologies is provided forexemplary purposes and can be derived and modified according by controloperators. In other words, the specific pixel measurements that controlthe adjustment of the adaptive parameter display (i.e., the values: “SDRreference range”, “SDR peak white” “Max. dest.”; “Tonemap HDR Range”;and “HLG Monitor”) can be refined and are not necessarily limited to themethodologies described above. However, it should be appreciated thatthe values of the bins of the luminance histogram 400 are used tocontrol the position of these adaptive parameter displayed on the lightintensity curve adjustment interface shown in FIG. 5.

As further shown in FIG. 5, the user interface 132 generates a Béziercurve that projects the hybrid log-gamma (“HLG”) pixels in the HDRcontent that is then tone-mapped (i.e., converted) to SDR content. Forexample, in one aspect, the disclosed system and method uses theIC_(T)C_(P) color space for the received HDR content, which is capped at1,000 nits as shown in FIG. 5 (i.e., the vertical line of the HLGmonitor). In this aspect, the down-mapping process will be more stablethan conventional HDR to SDR conversion by using the YCrCb color space.

In general, it is noted that with Y, Cr, Cb, traditionally used forcolor video representation, luminance Y, is strongly coupled with theColor Difference components. Y, Cr, Cb was developed to provide constantLuminance, but not constant hue. ICtCp provides a way to have bothconstant luminance, but also lines of constant hue. The ICtCp has beendefined to support the transformation of REC 2020 primaries, using boththe Dolby PQ and HLG non-linear companded intensity curves. I is theblack-white intensity aspect. Ct is the Color Tritan signal based onyellow blue, and Cp is the Color Protan signal, based on red-green.Mapping of RGB to the ICtCp color space is described in the Dolby WhitePaper on ICtCp, version 7.2, (available atwww.dolby.com/us/en/technologies/dolby-vision/ictcp-white-paper.pdf),the contents of which are hereby incorporated by reference. The mappingof RGB to YCrCb is known, and as these are linear sets of equations, anycan be mapped to the other. With ICtCp, the intensity can be adjustedwith little impact on the color, therefore luminance histogram can beused to evaluate the best setting for intensity, and then tone mappingcan occur to correct the color space. There will be some iteration, butbecause the effects of adjusting one or the other, are decoupled, thereis little interaction. Accordingly, the disclosed system and methodprovides a stable solution. With YCrCb, adjustments to Y significantlychange Cr or Cb, and then adjusting these would in turn be coupled to Y.As such, the process is much more circular than using a conventionalcolor mapping system.

Moreover, according to the exemplary aspect, the user interface isprovided with one or more control knobs (e.g., physical knob, touchscreen interface or the like) that is configured to enable the operatorto adjust the position (relative to the parameters) of the intensitycurve for the down-mapping process. That is, the user interface isconfigured such that by using the control knob, the operator (e.g.,camera shading operator) can adjust the projection/position of theBézier curve relative to the parameter values to control theillumination levels for the SDR content. Based on the positioning of theBézier curve as selected, the input and output luminance values are setaccording to the Bézier curve's positional relationship to the X and Yaxes of the light intensity curve adjustment interface. Once theintensity for the SDR (i.e., the output luminance) is set by theoperator, the system is further configured to perform color spacemapping to generate the output video production signal in SDR format.

It should be appreciated that according to the exemplary aspect, theshape of the Bézier curve can be used to best match the end ranges ofthe mapping process, essentially providing a piece-wise function totransition from the linear conversion range, to the end points, in asmooth way. This piecewise function is applied to the pixel values tocalculate the new mapped values, with the pixel values being representedin the ICtCp space.

Therefore, according to the exemplary aspects, the operator isadvantageously provided with a plurality of user interfaces for livevideo production that enable the operator to more actively manage andcontrol the HDR to SDR conversion to improve final picture quality withbetter picture illumination and contrast from an end consumptionperspective.

FIG. 6 illustrates a flow diagram for a method 600 for controllingluminance during video production and broadcast, in accordance withexemplary aspects of the present disclosure.

As shown, initially at 602, the method is initiated in which an HDRvideo capture device such as an HDR camera captures HDR video content,e.g., video content 104 at step 604. As described above, the capturedcontent is then provided to the image processor 110 shown in FIG. 1 forconversion into SDR content (or other desired image format).

At 606, the region selector 122 shown in FIG. 1 receives a selection ofone or more regions (e.g., region 300 shown in FIG. 3) of a frame ofcontent 104 from a user interface. In exemplary aspects, the region isselected by an operator of the system 100. In another exemplary aspect,the region may be selected automatically based on predefined user orprogram parameters specifying the specific region of pixels. It is againreiterated that in accordance with another exemplary embodiment, theentire frame of the content 104 is analyzed and thus step 606 isconsidered an optional step.

In either case, the method proceeds to 608, where the histogramgenerator 112 measures the luminance of each pixel contained within theregion selected by the operator. In some aspects, the operator mayselect several regions, and the histogram generator 112 measures theluminance of the pixels in some or all of the selected regions.

The method proceeds to 610, where the histogram generator 112 generatesa histogram by grouping each pixel based on light intensity into aparticular bin. An example of such a histogram is shown in FIG. 4,luminance histogram 400 and described above.

At step 612, a user interface is displayed to an operator of the system100, such as user interface 130 and 131 shown in FIGS. 3-4. The userinterface may display a frame of the content overlaid with the generatedhistogram based on the selected region. In exemplary aspects, this userinterface can be displayed as an initial step in the method, allowing auser to select a region via this interface, and dynamically regeneratingthe histogram each time the selection is modified.

At step 614, a user interface generator 118 displays a second userinterface (e.g., user interface 132) to the operator of the system 100that contains a light intensity curve, along with several otheradjustable parameters. The first and second user interfaces may bedisplay concurrently on a same screen according to an exemplary aspect.Moreover, the operator may adjust each of these parameters shown in thesecond user interface, which will then dynamically regenerate the lightintensity curve based on the adjustments. As further described above,the second user interface provides the ability to convert the lightintensity range of the HDR content to a range that is suitable fordisplay on SDR endpoints, and the operator may adjust each of theparameters according to the algorithms described above.

Once the adjustments are complete, the method proceeds to step 616 wherethe luminance controller convers the captured content 104 into aluminance range based on the luminance defined by the light intensitycurve from step 612. At 618, the converted content is encoded into SDRcontent (e.g., SDR content 141) and broadcast to endpoint devices, e.g.,display 150. According to an exemplary aspect, the method shown in FIG.6 may be continuously repeated as part of a loop so that the user, forexample, can continuously adjust the parameters as part of a feedbackloop, for example.

FIG. 7 is a block diagram illustrating a computer system 20 on whichaspects of systems and methods for controlling luminance during videoproduction and broadcast may be implemented in accordance with anexemplary aspect. It should be noted that the computer system 20 cancorrespond to the any components of the system 100. The computer system20 can be in the form of multiple computing devices, or in the form of asingle computing device, for example, a desktop computer, a notebookcomputer, a laptop computer, a mobile computing device, a smart phone, atablet computer, a server, a mainframe, an embedded device, and otherforms of computing devices.

As shown, the computer system 20 includes a central processing unit(CPU) 21, a system memory 22, and a system bus 23 connecting the varioussystem components, including the memory associated with the centralprocessing unit 21. The system bus 23 may comprise a bus memory or busmemory controller, a peripheral bus, and a local bus that is able tointeract with any other bus architecture. Examples of the buses mayinclude PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA,I²C, and other suitable interconnects. The central processing unit 21(also referred to as a processor) can include a single or multiple setsof processors having single or multiple cores. The processor 21 mayexecute one or more computer-executable codes implementing thetechniques of the present disclosure. The system memory 22 may be anymemory for storing data used herein and/or computer programs that areexecutable by the processor 21. The system memory 22 may includevolatile memory such as a random access memory (RAM) 25 and non-volatilememory such as a read only memory (ROM) 24, flash memory, etc., or anycombination thereof. The basic input/output system (BIOS) 26 may storethe basic procedures for transfer of information between elements of thecomputer system 20, such as those at the time of loading the operatingsystem with the use of the ROM 24.

The computer system 20 may include one or more storage devices such asone or more removable storage devices 27, one or more non-removablestorage devices 28, or a combination thereof. The one or more removablestorage devices 27 and non-removable storage devices 28 are connected tothe system bus 23 via a storage interface 32. In an aspect, the storagedevices and the corresponding computer-readable storage media arepower-independent modules for the storage of computer instructions, datastructures, program modules, and other data of the computer system 20.The system memory 22, removable storage devices 27, and non-removablestorage devices 28 may use a variety of computer-readable storage media.Examples of computer-readable storage media include machine memory suchas cache, SRAM, DRAM, zero capacitor RAM, twin transistor RAM, eDRAM,EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or othermemory technology such as in solid state drives (SSDs) or flash drives;magnetic cassettes, magnetic tape, and magnetic disk storage such as inhard disk drives or floppy disks; optical storage such as in compactdisks (CD-ROM) or digital versatile disks (DVDs); and any other mediumwhich may be used to store the desired data and which can be accessed bythe computer system 20.

The system memory 22, removable storage devices 27, and non-removablestorage devices 28 of the computer system 20 may be used to store anoperating system 35, additional program applications 37, other programmodules 38, and program data 39. The computer system 20 may include aperipheral interface 46 for communicating data from input devices 40,such as a keyboard, mouse, stylus, game controller, voice input device,touch input device, or other peripheral devices, such as a printer orscanner via one or more I/O ports, such as a serial port, a parallelport, a universal serial bus (USB), or other peripheral interface. Adisplay device 47 such as one or more monitors, projectors, orintegrated display, may also be connected to the system bus 23 across anoutput interface 48, such as a video adapter. In addition to the displaydevices 47, the computer system 20 may be equipped with other peripheraloutput devices (not shown), such as loudspeakers and other audiovisualdevices

The computer system 20 may operate in a network environment, using anetwork connection to one or more remote computers 49. The remotecomputer (or computers) 49 may be local computer workstations or serverscomprising most or all of the aforementioned elements in describing thenature of a computer system 20. Other devices may also be present in thecomputer network, such as, but not limited to, routers, networkstations, peer devices or other network nodes. The computer system 20may include one or more network interfaces 51 or network adapters forcommunicating with the remote computers 49 via one or more networks suchas a local-area computer network (LAN) 50, a wide-area computer network(WAN), an intranet, and the Internet. Examples of the network interface51 may include an Ethernet interface, a Frame Relay interface, SONETinterface, and wireless interfaces.

Aspects of the present disclosure may be a system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present disclosure.

The computer readable storage medium can be a tangible device that canretain and store program code in the form of instructions or datastructures that can be accessed by a processor of a computing device,such as the computing system 20. The computer readable storage mediummay be an electronic storage device, a magnetic storage device, anoptical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination thereof. Byway of example, such computer-readable storage medium can comprise arandom access memory (RAM), a read-only memory (ROM), EEPROM, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),flash memory, a hard disk, a portable computer diskette, a memory stick,a floppy disk, or even a mechanically encoded device such as punch-cardsor raised structures in a groove having instructions recorded thereon.As used herein, a computer readable storage medium is not to beconstrued as being transitory signals per se, such as radio waves orother freely propagating electromagnetic waves, electromagnetic wavespropagating through a waveguide or transmission media, or electricalsignals transmitted through a wire.

Computer readable program instructions described herein can bedownloaded to respective computing devices from a computer readablestorage medium or to an external computer or external storage device viaa network, for example, the Internet, a local area network, a wide areanetwork and/or a wireless network. The network may comprise coppertransmission cables, optical transmission fibers, wireless transmission,routers, firewalls, switches, gateway computers and/or edge servers. Anetwork interface in each computing device receives computer readableprogram instructions from the network and forwards the computer readableprogram instructions for storage in a computer readable storage mediumwithin the respective computing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembly instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language, and conventional procedural programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a LAN or WAN, or theconnection may be made to an external computer (for example, through theInternet). In some aspects, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present disclosure.

In various aspects, the systems and methods described in the presentdisclosure can be addressed in terms of modules. The term “module” asused herein refers to a real-world device, component, or arrangement ofcomponents implemented using hardware, such as by an applicationspecific integrated circuit (ASIC) or FPGA, for example, or as acombination of hardware and software, such as by a microprocessor systemand a set of instructions to implement the module's functionality, which(while being executed) transform the microprocessor system into aspecial-purpose device. A module may also be implemented as acombination of the two, with certain functions facilitated by hardwarealone, and other functions facilitated by a combination of hardware andsoftware. In certain implementations, at least a portion, and in somecases, all, of a module may be executed on the processor of a computersystem (such as the one described in greater detail in FIG. 7, above).Accordingly, each module may be realized in a variety of suitableconfigurations, and should not be limited to any particularimplementation exemplified herein.

In the interest of clarity, not all of the routine features of theaspects are disclosed herein. It would be appreciated that in thedevelopment of any actual implementation of the present disclosure,numerous implementation-specific decisions must be made in order toachieve the developer's specific goals, and these specific goals willvary for different implementations and different developers. It isunderstood that such a development effort might be complex andtime-consuming, but would nevertheless be a routine undertaking ofengineering for those of ordinary skill in the art, having the benefitof this disclosure.

Furthermore, it is to be understood that the phraseology or terminologyused herein is for the purpose of description and not of restriction,such that the terminology or phraseology of the present specification isto be interpreted by the skilled in the art in light of the teachingsand guidance presented herein, in combination with the knowledge of theskilled in the relevant art(s). Moreover, it is not intended for anyterm in the specification or claims to be ascribed an uncommon orspecial meaning unless explicitly set forth as such.

The various aspects disclosed herein encompass present and future knownequivalents to the known modules referred to herein by way ofillustration. Moreover, while aspects and applications have been shownand described, it would be apparent to those skilled in the art havingthe benefit of this disclosure that many more modifications thanmentioned above are possible without departing from the inventiveconcepts disclosed herein.

What is claimed:
 1. A system for controlling luminance of video contentduring video production, the system comprising: at least one cameraconfigured to capture video content in a first imaging range; ahistogram generator configured to measure a respective luminance levelfor each of a plurality of pixels in at least one frame of the capturedvideo content, and to generate a luminance histogram based on themeasured luminance levels of the plurality of pixels in the at least oneframe; a first user interface configured to display the at least oneframe of the captured video content and the generated luminancehistogram as an overlay on the displayed at least one frame; a seconduser interface configured to display a light intensity curve relative toa plurality of parameters that include at least one of a standarddynamic range (SDR) reference range, an SDR peak white, a tonemapMax-Destination, a tonemap high dynamic range (HDR) range, and a hybridlog-gamma (HLG) monitor; a luminance controller configured to convertthe captured video content in the first imaging range to broadcast videocontent in a second imaging range that is different than the first imagerange and has a luminance based on output luminance values of the lightintensity curve; and a broadcast controller configured to encode thebroadcast video content in the second imaging range for distribution toat least one content consuming device for display thereon, wherein thesecond user interface displays the plurality of parameters and isfurther configured to adjust the light intensity curve in response to auser adjustment of a value of at least one of the plurality ofparameters to set the output luminance values for the luminancecontroller to convert the captured video content in the first imagingrange to the broadcast video content in the second imaging range.
 2. Thesystem of claim 1, wherein the first imaging range is an HDR and thesecond imaging range is an SDR.
 3. The system of claim 2, wherein thesecond user interface is configured to receive a selection of one ormore of the following HDR to SDR conversion methodologies for convertingthe captured video content in the first imaging range to broadcast videocontent in the second imaging range: (1) High Mid-tone DetailEnhancement; (2) Highlight Detection; (3) Strong Highlight Enhancement;and (4) Gamut excursion correction.
 4. The system of claim 1, whereinthe captured video content is live video content.
 5. The system of claim1, wherein the plurality of parameters displayed in the second userinterface include each of the SDR reference range, the SDR peak white,the tonemap Max-Destination, the tonemap HDR range, and the HLG monitor.6. The system of claim 1, wherein the histogram generator is configuredto generate the luminance histogram that includes 16 separate luminanceranges with a first range of the ranges beginning at less than 0.5 nitsand a last range of the ranges being for pixels in the 8192-10000 nitsrange.
 7. The system of claim 6, wherein the histogram generator isconfigured to generate the luminance histogram by precomputing a onedimensional lookup table, wherein a key of the one dimensional lookuptable is X bits.
 8. A system for controlling luminance of video contentduring video production, the system comprising: at least one cameraconfigured to capture video content in a first imaging range; ahistogram calculator configured to evaluate luminance in the capturedvideo content, and generate a luminance histogram for the captured videocontent; a user interface generator configured to: generate a first userinterface displaying the captured video content overlaid with thegenerated luminance histogram, and generate a second user interfacedisplaying a light intensity curve configured with parameters forconverting the captured video content from the first imaging range intoa second imaging range, with the parameters including at least one of astandard dynamic range (SDR) reference range, an SDR peak white, atonemap Max-Destination, a tonemap high dynamic range (HDR) range, and ahybrid log-gamma (HLG) monitor; and a luminance controller configured toconvert the captured video content into the second imaging range basedon the parameters of the light intensity curve, wherein the second userinterface displays the parameters and is further configured to adjustthe light intensity curve in response to a user adjustment of a value ofat least one of the parameters to control the luminance controller toconvert the captured video content into the second imaging range.
 9. Thesystem of claim 8, further comprising a broadcast controller configuredto encode the converted video content in the second imaging range fortransmitting to one or more endpoint devices.
 10. The system of claim 8,wherein the second user interface includes at least one control inputconfigured adjust a position of the light intensity curve relative tothe parameters to set output luminance values for the luminancecontroller to convert the captured video content in the first imagingrange to the second imaging range.
 11. The system of claim 8, whereinthe first imaging range is an HDR and the second imaging range is anSDR.
 12. The system of claim 11, wherein the second user interface isconfigured to receive a selection of one or more of the following HDR toSDR conversion methodologies for converting the captured video contentin the first imaging range to broadcast video content in the secondimaging range: (1) High Mid-tone Detail Enhancement; (2) HighlightDetection; (3) Strong Highlight Enhancement; and (4) Gamut excursioncorrection.
 13. The system of claim 8, wherein the captured videocontent is live video content.
 14. The system of claim 8, wherein theparameters displayed in the second user interface include each of theSDR reference range, the SDR peak white, the tonemap Max-Destination,the tonemap HDR range, and the HLG monitor.
 15. The system of claim 8,wherein the histogram calculator is configured to generate the luminancehistogram that includes 16 separate luminance ranges with a first rangeof the ranges beginning at less than 0.5 nits and a last range of theranges being for pixels in the 8192-10000 nits range, and wherein thehistogram calculator is configured to generate the luminance histogramby precomputing a one dimensional lookup table, wherein a key of the onedimensional lookup table is X bits.
 16. The system of claim 8, whereinthe luminance controller is further configured to dynamically adjust thesecond imaging range based on the parameters of the light intensitycurve.
 17. The system of claim 8, wherein the user interface generatoris further configured to generate the first and second user interfacebased on endpoint device display information.
 18. A system for settingluminance of video content, the system comprising: a histogram generatorconfigured to generate a luminance histogram based on measured luminancevalues for at least one region in video content in a first imagingrange; a user interface generator configured to generate at least oneuser interface configured to display the generated luminance histogramand a light intensity curve with associated parameters for convertingthe video content from the first imaging range to a second imagingrange; and a luminance controller configured to convert the videocontent to the second imaging range based on settings of the associatedparameters of the light intensity curve, wherein the at least one userinterface is further configured to adjust a position of the lightintensity curve in response to a user adjustment of a value of at leastone of the associated parameters, and wherein the parameters displayedin the at least one user interface include a standard dynamic range(SDR) reference range, an SDR peak white, a tonemap Max-Destination, atonemap high dynamic range (HDR) range, and a hybrid log-gamma (HLG)monitor.
 19. The system of claim 18, further comprising a broadcastcontroller configured to encode the converted video content in thesecond imaging range for transmitting to one or more endpoint devices.20. The system of claim 18, wherein the first imaging range is an HDRand the second imaging range is an SDR.
 21. The system of claim 20,wherein the at least one user interface is further configured to receivea selection of one or more of the following HDR to SDR conversionmethodologies for converting the captured video content in the firstimaging range to the second imaging range: (1) High Mid-tone DetailEnhancement; (2) Highlight Detection; (3) Strong Highlight Enhancement;and (4) Gamut excursion correction.
 22. The system of claim 18, whereinthe captured video content is live video content.
 23. The system ofclaim 18, wherein the histogram generator is configured to generate theluminance histogram that includes 16 separate luminance ranges with afirst range of the ranges beginning at less than 0.5 nits and a lastrange of the ranges being for pixels in the 8192-10000 nits range, andwherein the histogram calculator is configured to generate the luminancehistogram by precomputing a one dimensional lookup table, wherein a keyof the one dimensional lookup table is X bits.
 24. The system of claim18, wherein the luminance controller is further configured todynamically adjust the second imaging range based on the parameters ofthe light intensity curve.
 25. The system of claim 18, wherein the userinterface generator is further configured to generate the at least oneuser interface based on endpoint device display information.